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The Relationship Between Usage of the Algebra Nation Tutoring Program and Test 


Performance for Students Who Fail and Retake the End-of-Course Exam 


Abstract 
Although the use of technology in the K12 classroom has been shown to have a positive impact, 
research on the use of open education resources (OER) is relatively limited, especially research 
with a focus on low-achieving students. Using secondary data, the present study examines the 
relationship between usage of Algebra Nation, a popular self-guided system that provided 
instructional videos and practice problems, and the performance of students who had failed the 
state-administered Algebra I end-of-course (EOC) assessment the previous year. Indicators of 
usage of Algebra Nation consisted of logins, video views, and practice questions answered in the 
Test Yourself module. Path analyses and logistic regressions were used to evaluate relationships 
between Algebra Nation usage indicators and algebra scores, controlling for number of absences, 
free/reduced lunch eligibility, Hispanic/Latino origin, race and gender. The results indicate that 
higher levels of logins, video views and practice questions answered were related to higher 
scores when the students re-took the assessment. Logins and practice questions were also related 
to increases in odds of passing the Algebra I EOC assessment, but not video views. The results 
suggest that there may be benefits to technology use in the form of an OER adopted by students 
and teachers on an informal basis, and link self-regulated learning strategies to student 


achievement. 
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The relationship between Algebra Nation usage and highstakes test performance for struggling students 


Recent research suggests that educational technology has become thoroughly integrated 
into K12 classrooms. Students and teachers no longer make a distinction between learning 
experiences that do or do not utilize technology, and most students now have access to the 
Internet at home and at school (NAEP, 2015; National Forum on Education Statistics, 2015). 
Education technology is now recognized by educators, parents, and policy makers for its 
potential to improve student learning outcomes. Recent meta-analyses have documented a 
generally positive effect associated with the use of education technology in the classroom 
(Cheong & Slavin, 2013; Kulik & Fletcher, 2016; Steenbergen-Hu & Cooper, 2013). 

As technology-based resources have become increasingly established in the K12 
classroom, there has been a greater consideration given to the nature of the technology, referring 
to its source, design and how it is intended to be used. Cheung and Slavin (2013) considered 
three types of education technology: supplemental computer-based resources to provide students 
with additional practice opportunities, computer-based learning management systems designed to 
assess and track student progress, and comprehensive models in which learning activities that 
take place at the computer are embedded into a theory-based curriculum. Reviews conducted by 
Steenbergen-Hu and Cooper (2013b) and Kulik and Fletcher (2016) specifically considered 
studies of intelligent tutoring systems, referring to computer-based systems that continually adapt 


instruction to students’ individual strengths and weaknesses (Koedinger et al., 1997). These 
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systems have typically been designed and developed by educational publishers or researchers 
based on theories of how people learn. Most of these types of resources also include clear 
guidelines for how the technology is to be used and are often supported by professional 
development training for teachers. Often, states or districts specify what resources may be used 
and there is a formal review and adoption process for technology products (Means et al., 2016). 
An alternative form of technology includes “open educational resources” (OERs; UNESCO & 
Commonwealth of Learning, 2011). Many are user-generated, shared freely on the web, and 
available for use on an ad hoc basis. Such resources differ from the education technology 
platforms considered by Cheung and Slavin (2013), Steenbergen-Hu and Cooper (2013b) and 
Kulik and Fletcher (2016) in that they are not necessarily grounded in theories of learning, used 
as part of a comprehensive curriculum, or accompanied by professional development for 
instructors. For example, the popular Khan Academy website began in 2006 as a collection of 
screen-cast videos made by Sal Khan who posted the videos on YouTube to share solutions to 
math problems to a young relative who lived across the country (Thompson, 2011). The site 
quickly attracted users who visited on an informal, drop-in basis. The site now has expanded to 
include instructional videos on hundreds of topics and is visited by millions of viewers per 
month (Phillips & Cohen, 2015). 

OERs have proliferated in the last 15 years, and there has been a corresponding increase 
in interest in knowing how they are being used and what their impact is on students (Means, 
Anderson & Thomas, 2013; Watson, Pape, Murin, Gemin & Vashaw, 2014). The use of OERs is 
driven informally by students and teachers themselves rather than being a formal part of a 
curriculum (Olcott Jr., 2012). There is a current gap in the literature on strategies to support 


students to use OER, but there have been attempts to provide a framework for development of 
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strategies (Kim, 2018). In order to design strategies for students to use OER, it is critical to 
investigate their impact on student achievement. Because the use of OERs is often casual and 
widely varied across users, researchers are adopting new approaches to explore their impact 
(Koedinger et al., 2015). One strategy is to use digital records documenting students’ use of an 
open resource and then investigating variations in learning outcomes in relation to use 
(Koedinger et al., 2015, 2016; Means et al., 2013). Although this approach will not support 
causal conclusions, it is an initial step towards assessing if students who have used an OER may 
have received some corresponding benefit from it. For example, in the case of Khan Academy, 
researchers used records of student logins to identify frequent and less frequent users of the SAT 
assessment practice resources. Frequent users achieved higher scores on the SAT assessment 
than would have been predicted on the basis of their PSAT score (Murphy, Gallagher, Krumm, 
Mislevy & Hafter, 2014). Although conclusions were limited by the lack of a comparison group 
and by the focus on high achieving students, the initial evidence was promising enough to 
support more traditional experimental research incorporating random assignment (currently 
underway; see Schneider & Hauk, 2014). 

Systematic evaluations of the relationship between the use of OERs for the K12 
classroom and student achievement are starting to appear in the literature. For example, 
Roschelle, Feng, Murphy and Mason (2016) conducted a cluster randomized trial of the 
ASSISTments platform, which provides online homework assistance and feedback to students 
and reports to teachers. They a significant difference of 0.18 standard deviations in the 
mathematics performance of students in schools using ASSISTments as compared to schools 
using traditional homework. The present study contributes to this literature by reporting the 


initial results associated with a recent example of an OER widely-used in the state of Florida: 
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Algebra Nation (AN). AN is a tutoring system designed to prepare students for the state- 
administered Algebra I end-of-course assessment (EOC), which is a high-stakes test in Florida 
required for high school graduation. AN was originally launched in 2013 as a free stand-alone 
website by a partnership between a university research center and a small tutoring business. 
Although not strictly an OER because it was not released under an open license (UNESCO & 
Commonwealth of Learning, 2011), AN shares the following OER characteristics: 1) Students 
have autonomy to use it away from formal curricula; 2) It allows self-paced/regulated learning; 
3) It is available on an ad-hoc basis. Also, AN functions as an OER within Florida because every 
student and teacher has free access to it. AN quickly attracted users through word-of-mouth and 
by 2016, records in the AN database documented that there were some users in every district in 
the state. However, usage patterns varied widely, making it difficult to determine whether AN 
use was actually associated with increased assessment scores as intended by the developers. 

AN includes instructional videos and practice problems that resemble those on the actual 
EOC assessment. Students can also download a workbook of practice problems from AN, and 
the videos address solving problems in the workbook. Moreover, teachers are able to request 
paper workbooks for their students that contain all the practice problems. The platform has some 
similarities to the original Khan Academy in that it is self-guided and freely available to any 
student or teacher in the state. AN has about 100 videos with scope and sequence aligned with 
the Florida standards, with about 10 videos for each of 9 algebra topics. There are multiple 
versions of each video with different tutors, with different durations and levels of detail that the 
content is covered. The differences in depth of coverage is highlighted by a sliding bar varying 
from “review” to “in-depth” next to the tutor’s name and picture. Therefore, students can choose 


tutors with the presentation style that they find most appealing. The “Test Yourself” practice 
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module is designed to look like the actual EOC assessment, including an integrated calculator 
and clock/timer. Also, there is an item pool with about 100 items for each of nine algebra topics. 
The topics covered by AN are: 1) Expressions, 2) Equations and inequalities, 3) Introduction to 
functions, 4) Linear equations, functions and inequalities, 5) quadratic functions, 6) Exponential 
functions, 7) Summary of functions, 8) One-variable statistics, 9) Two-variable statistics. The 
items include multiple formats: equation response, numeric fill in the blank, drop-down select, 
multi-select, and multiple choice. There are both one-step and multiple-step items and some have 
multiple solution paths. When students select to take the Test Yourself practice module for one 
algebra topic, the system randomly selects 10 items to administer. For each item, there is a 
solution video, and a review video for the topic. There is a pre-algebra remediation tool available 
with a separate set of videos and assessment questions. Students have access to the Algebra 
Wall, which is a discussion forum monitored by tutors who assist students with their questions. 
However, AN is different from intelligent tutoring systems in that it does not prompt students for 
steps or provide hints. Also, the process of navigation through the system is completely driven 
by the student. When the student answers a question incorrectly, it only points out that the 
answer is incorrect and provides the solution video and a review video of the concepts, but it 
does not provide an elaboration on the reasons for an answer being incorrect. Also, practice 
questions that have multiple steps are graded as correct or incorrect, and there is no partial credit 
or feedback on which steps were incorrect. 

Initial research (Leite, 2017; Mitten, Collier & Leite, 2016) provided preliminary 
evidence that the use of AN in schools is associated with increased mean EOC assessment scores 
and increased passing rates. More specifically, Leite (2017, p.143) performed a generalized 


propensity score analysis of school level data to estimate the effects of number of AN logins per 
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examinee in each school on the school’s mean score on the EOC assessment. He found a small 
but statistically significant effect, with an increase in 10% in logins per examinee corresponding 
to predicted increase in mean EOC scores of 0.063. Mitten, Collier & Leite (2016) performed a 
latent class analysis of six school-level indicators of AN usage: 1) average number of student 
logins, 2) average number of student videos viewed, 3) average number of teacher logins, (4) 
average number of teacher videos viewed, 5) ratio of total videos viewed and the number of 
ordered workbooks, 6) ratio of total number of logins and the numbered of ordered workbooks. 
They found that schools in the sample could be clustered into 3 classes of different level of usage 
of AN, and that schools in classes with higher level of usage also had mean EOC assessment 
scores and higher passing rates. 

The current study contains the first student-level analysis to focus specifically on re- 
takers. In contrast to the initial research on Khan Academy which focused on high-achieving, 
college-bound students, here the focus was on those students most at risk of failure. Successful 
completion of high school algebra courses has been identified as an important milestone for 
college readiness, college enrollment, attainment of a college degree and future employment. 
Unfortunately, algebra courses typically have a high failure rate. Failure means that many 
students must retake algebra (Fong, Jaquet & Finkelstein, 2014). However, the prospects for re- 
takers are grim because when students fail Algebra once, they are likely to fail it again (Ham & 
Walker, 1999, Helfand, 2006; Waterman, 2010). In 2017, 73% of students who took the Florida 
EOC assessment for a second time (or more) failed again. 

The need to repeat algebra is costly for the education system and for the student (Fong, 
Jaquet, & Finkelstein, 2014). Recent research suggests that educational technology could 


potentially be an effective and cost-efficient approach to algebra instruction (Bishop, 2006; Jung, 
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2003; Taplin, Ross, Kerr & Brown, 2013). Studies of online programs have shown benefits 
relative to traditional classroom instruction (Cavanaugh, Gillan, Bosnick, Hess & Scott, 2008; 
Hagerty & Smith, 2005; Heid & Blume, 2008; Heppen et al., 2012; Jaciw, Megan & Boya, 2012; 
Pane, Griffin, McCaffrey & Karam, 2014; Patrick, & Powell, 2009; Rakes, Valentine, McGatha 
& Ronau, 2010; Roschelle et al., 2010; Shaw, Jean & Peck, 1997). However, this research 
generally considered the impact of traditional technology aligned with a district curriculum, 
rather than OERs whose usage was determined by students and teachers on an informal basis. 
Also, in the case of students who have already failed algebra once or more and are therefore at 
high risk for future failure, the potential of educational technology is less clear. One study of 
credit recovery courses found that students who took a face-to-face course were more successful 
than those who took the online version (Heppen et al., 2016). Thus, it was not clear that casual 
use of an OER, however popular, would actually be associated with a detectable effect in algebra 
achievement of students who were struggling with the material. 

The goal of the present study is to evaluate the relationships between usage of resources 
within AN by students with a history of failure on the EOC assessment and the scores these 
students obtain when they re-take the assessment. The conceptual model for the study is shown 
in Figure 1. A similar analytic strategy to that used in the case of Khan Academy (Murphy, 
Gallagher, Krumm, Mislevy & Hafter, 2014) was adopted: students in the AN database who had 
already failed the assessment once were identified, records of their use of AN were compiled, 
and relationships between AN use and EOC assessment scores were estimated. The analysis also 
included an investigation into the specific activities (i.e., watching videos, completing practice 
problems) within the AN platform that appeared to be most helpful. Based on prior work 


showing greater benefits of active practice, one might expect that doing practice problems would 
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have a stronger positive relationship with algebra scores compared to viewing instructional 
videos (Koedinger et al., 2016). Problem-solving practice benefits the learner by providing 
feedback and also appears to be a more efficient use of study time than viewing videos 
(Koedinger et al., 2015). However, evidence to support the “do-er” effect was found in studies of 
college students and those enrolled in college-level MOOCs, that is, students who may be 
assumed to have some level of motivation to learn as well as reasonable study skills. It was not 
clear whether a similar benefit of practice would be observed in a sample of very low-achieving 


students who were facing the risk of not qualifying for high school graduation. 


INSERT FIGURE 1 ABOUT HERE 


Research Questions 

1. For the students who failed the EOC assessment in 2014/2015, was there a 
relationship between usage of the Algebra Nation system in 2014/2015 and the 
students’ scores? 

2. For the students who failed the EOC assessment in 2014/2015, does higher usage of 
the Algebra Nation system in 2015/2016 predict higher scores and passing rates when 
they retook the EOC assessment in 2015/2016? 

3. For the students who failed the Algebra EOC assessment in 2014/2015, do video 


views and completion of practice problems in the AN platform have different 
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relationships with student scores and passing rates when they retook the EOC 


assessment in 2015/2016? 


For the first research question, the hypothesis is that even though all students in the 
sample failed the EOC assessment in 2014/2015, the students who used Algebra Nation more 
frequently performed better than those that used it less frequently, after controlling for 
established academic risk factors (i.e., free/reduced lunch eligibility, days absent) and 
demographic factors (i.e., ethnicity, Hispanic origin, and gender). For the second research 
question, the hypothesis is that using AN more frequently in 2015/2016 is associated with re- 
takers obtaining higher EOC assessment scores and higher passing rates, controlling for the 
students’ 2014/2015 EOC score, free/reduced lunch eligibility, days absence, ethnicity, Hispanic 
origin, and gender. For the third research question, the hypothesis is that completion of practice 
problems will have stronger associations with EOC assessment scores in both 2014/2015 and 
2015/2016 than video views. 

Method 
Sample 
The sample for this study included all students in Florida public schools who satisfied the 
following conditions: (a) they took the EOC assessment for the first time in 2014-2015 and did 
not obtain a passing score; (b) they logged into Algebra Nation at least once in 2014-2015; and 
(c) they re-took the assessment in 2015/2016. The total sample size was 3,987 and included 7" 
to 11" grade students. The frequencies and percentages by grade level, gender, race, ethnicity 
and free/reduced lunch eligibility are shown in Table 1. The re-takers included in this study were 
more likely to be not-white (48%), eligible for free/reduced lunch (79%) and of Hispanic/Latino 


origin (32%) as compared to students taking the EOC assessment for the first time in 2015-2016 


First published on May 23 2019 
11 


(34% not-white, 58% eligible for free/reduced lunch, 29% Hispanic/Latino origin). The 
proportion of students who were female was similar for re-takers and first-time-takers (55%). 
Data Sources 
Two datasets were combined for this analysis. The indicators of usage were obtained 

from Study Edge, the developer of the AN platform. The EOC assessment scores and 
demographic information of students were obtained from the Florida Department of Education. 
The combined dataset contained observations across two years (2014/2015 and 2015/2016) for 
each student. Students were clustered in 403 schools in the 2014/2015 school year, and 339 
schools in the 2015/2016 school year. The average number of students who failed the EOC 
assessment per sampled school was 9.89. 
Measures 

Data about students’ usage of AN during the school year, as indicated by the total number of 
logins, video views, and use of the Test Yourself practice module, was provided by the AN 
platform developer. Each usage indicator was binned into one of three frequency categories: 
fewer than five logins/video views/Test Yourself question responses, between five and 30 
logins/video views/Test Yourself question responses, and more than 30 logins/video views/Test 
Yourself question responses. This categorization of the activity counts was performed because 
the total counts of usage were strongly positively skewed and non-linearly related to the 
outcome. To maintain evidence of construct validity for the usage indicators in this study, the 
boundaries of the categories were chosen in consultation with the AN development team to 
reflect students that may have looked at AN but did not use it in a meaningful way (i.e., fewer 


than five usage indicators), students who had moderate engagement with AN (i.e., between five 
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and 30 uses), and frequent users (i.e. more than 30 counts). Sample sizes for the use categories 
are included in Table 1. 

Multiple demographic variables were obtained from the Florida Department of Education 
and matched by student ID to the AN data system. The specific demographic variables were the 
number of absences in each of the 2014/2015 and 2015/2016 school years, free/reduced lunch 
status as a proxy for socioeconomic status, gender, white race, and Hispanic/Latino origin of 
students. 

Algebra Scores. The outcome measure was the student’s score on the EOC assessment at 
the end of each of the two school years (2014/2015 and 2015/2016). The EOC assessment is a 
high-stakes standardized test administered in Florida to all students enrolled in algebra courses 
aligned with the Florida standards, such as Algebra I and Algebra I Honors. Students are 
required to obtain a specific score to pass the test and graduate from high school. The passing 
score is determined by Florida’s State Board of Education. The test is a computer-based 
assessment that includes both multiple choice and fill-in responses (Florida Department of 
Education, 2018). 

Data preparation 

Data from different sources were aligned and then identifying information was removed 
before the dataset was made available for the analysis. The dataset had missing values in the 
students’ demographic information and algebra scores. Initially, missing values were dealt with 
using listwise deletion by removing students who had missing EOC scores in 2014/2015 and 
2015/2016. Then, the dataset was filtered to obtain students who failed the EOC assessment by 
removing students who had scores equal to or greater than 497 (the cut score to pass the 


assessment) in 2014/2015. Because demographic variables used in this study were time- 
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invariant, except the days-absent variable, remaining missing values in these variables were 
handled by replacing missing information in the 2015/2016 year with available information in 


the 2014/2015 year. 


INSERT TABLE 1 ABOUT HERE 


Analysis 

We used two path analysis models and two logistic regression models to address the 
research questions. We created a path analysis model including AN login categories as predictors 
of EOC assessment scores, and another path analysis model with both categories of video views 
and Test Yourself module usage as predictors. Separate path analysis models were used because 
video views and Test Yourself module usage were expected to fully mediate the relationship 
between logins and the outcome, so including all indicators of usage in the same model would 
remove the relationships between logins and algebra scores. The implication of not including 
logins in the models that included video views and Test Yourself module usage is that if two 
students watched the same number of videos and responded to the same number of Test Yourself 
module questions, they were considered to have the same level of AN usage despite possible 
differences in the number of times they logged in to AN. The path analysis model for login 


categories as predictors of algebra scores was defined as: 
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Le = By + B,,L,; + ByL,; + BA); + B,,G, aE 5M; + BF, + BLY; + 5; * (2) 


In Equations | and 2, Y,, and Y,, are the student EOC assessment scores in 2014/2015 and 


2015/2016, respectively. The three login categories were used to create two dummy-coded 


variables per year, where L,,, and L,,, indicate five to 30 logins in 2014/2015 and 2015/2016, 
L,,;and L,,, indicate more than 30 logins for the same years, and the reference category is less 
than five logins. For both outcomes, the model controlled for the number of absences (i.e., A, 
and A,, in 2014/2015 and 2015/2016), gender (i.e., G,;= 1 indicates female), minority status (M, 
= | indicates white), free/reduced lunch status (1.e., F;=1 indicates eligible for free or reduced 
lunch), and Hispanic ethnicity (H,=1 indicates Hispanic student). The model also included the 
2014-2015 EOC assessment score as a predictor of the 2015-2016 score. The residuals at each 
year are e,,and e,,. The path analysis model for categories of video views and Test Yourself 


module answers as predictors of EOC assessment scores was defined as: 


Yi =Not+NMii t+NMiai + Nalin + Malia + Msi + NeGi + 1M + Nei + MoH +41: » (3) 


and 


Yop = Yo0 + Yoo + Yori + Yost ou: + Vogl oni + NsAu + NeG; + %o7Mi + Pog + YogX%i ten (4 
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In Equations 3 and 4, V,,, and V,,; indicate video views, and T,,,and T,,, indicate Test Yourself 
module responses in the five to 30 category for 2014-2015 and 2015-2016, respectively. The 
variables V,,,and V,,; indicate video views, and T,,; and T,,; indicate Test Yourself module 


responses in the greater than 30 category for 2014-2015 and 2015-2016. The remaining variables 
in Equations 3 and 4 are defined as in Equations 1 and 2. 

Before examining the path model parameter estimates, model fit statistics and indices 
were examined to determine whether the path models in Equations | to 4 fit the data well. We 
first examined the chi-square test to determine if there was exact fit, and then looked at fit 
indices to determine close fit based on Hu and Bentler’s (1999) recommended cutoff criteria of 
root mean squared error of approximation (RMSEA) < .06, Tucker Lewis index (TLI) = .95, 
Comparative Fit index (CFI) > .95, and standard root mean square residual (SRMR) < .08. 

To examine whether AN usage in 2015/2016 predicted passing the re-take of the EOC 
assessment, we used two logistic regression models. The first logistic regression model used 
dummy-coded login categories as predictors, while the second used dummy-coded video view 
and Test Yourself module response categories as predictors, controlling for the same covariates 


as in Equation 2 and 4. These two logistic regression models were defined as: 


In( p, /Q- P;)) = Boy + Bylo + Bylo + Box Agi + BusG + PosMi + Book + BY; (5) 


and 


In(p, /(- P;)) = Yr t+ Yaoi + VVnri + Yostari + Voatoai + Ns Aui + 


(6) 
V6Gi + Voy 5 + YogFi + YooXi- 
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where Pp, is the probability of passing the re-take of the EOC assessment in 2014/2015, and the 


predictors are defined as in Equations 2 and 4. 

All models were estimated with the Mplus 8.1 software (Muthén & Muthén, 2017) with 
robust maximum likelihood estimation with cluster robust standard errors (Stapleton, 2008) to 
account for the clustering effects of schools. The use of cluster robust standard errors is an 
alternative to multilevel modeling when students are nested within classrooms and schools, and 
is recommended when the focus of the analysis is solely on student-level variables (McNeish, 
Stapleton, & Silverman, 2017). In the path analysis models, login, video view, and Test Yourself 
activities in 2014/2015 have an indirect effect on EOC assessment scores in 2015/2016 through 
the mediation of scores in 2014/2015. The standard errors for these indirect effects were 
calculated using the delta method (Sobel, 1982) as implemented in Mplus 8.1. 

Results 
Path Analysis of the Relationship Between Algebra Nation Logins and Algebra Scores 

For the evaluation of fit of the path analysis model in Equations 1, the chi square test was 
not significant [i.e., X7(6, N = 3987) = 7.063, p = 0.315], indicating that the model had exact fit 
to the data. This conclusion was supported by fit indices indicating close fit, with CFI = .997, 
TLI = .991, RMSEA = .007, SRMR = .004. Thus, we concluded that the first path model fit the 
data well, and interpreted the model parameter estimates. 

Table 2 shows the standardized coefficients estimated for the first equation of the path 
analysis model (i.e., Equation 1), which predicts algebra scores in 2014/2015. We found that 
students who logged in between 5 and 30 times during 2014/2015 had EOC scores that were 


0.086 standard deviation units higher on average than students with fewer than five logins, after 
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controlling for free/reduced lunch status, days absent, race, Hispanic origin, and gender. Table 3 
shows the results for the prediction of Algebra EOC scores in 2015/2016, which corresponds to 
Equation 2 of the path analysis model. It shows that students who logged in between 5 and 30 
times during 2015/2016 had EOC scores that were 0.093 standard deviation units higher on 
average than students with fewer than five logins. Also, students who logged in more than 30 
times during 2015/2016 had algebra scores that were 0.195 standard deviation units higher on 
average than students with fewer than five logins, after controlling for covariates. Table 3 also 
shows that there was a significant indirect relationship between student logins in 2014/2015 and 
algebra scores in 2015/206, mediated by algebra scores in 2014/2015. More specifically, students 
who logged in between 5 and 30 times during 2014/2015 had a predicted 2015/2016 algebra 
score that was 0.024 standard deviations higher than those students with less than five logins. 

Results in Table 2 show that students of Hispanic or Latino origin had algebra scores in 
2014/2015 that were 0.147 standard deviation units lower than students not of Hispanic or Latino 
origin, controlling for other variables in the model. Students who were eligible for free/reduced 
lunch had algebra scores in 2014/2015 that were 0.179 standard deviation units lower than 
students who were not eligible for free/reduced lunch, controlling for other variables in the 
model. Also, white students had algebra scores in 2014/2015 that were 0.137 standard deviation 
units higher than non-white students, controlling for other variables in the model. 

Table 3 shows both direct and indirect relationships of demographic variables, which can 
be summed to obtain a total relationship. The indirect relationships occur because some of the 
demographic variables have direct relationships with the 2014/2015 algebra EOC scores, which 
in turn has a direct relationship with 2015/2016 algebra EOC scores. In 2015/2016, students who 


were eligible for free/reduced lunch had algebra scores that were 0.206 (0.157 directly and 0.049 
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indirectly) standard deviation units lower than students who were not eligible for free/reduced 
lunch. Hispanic or Latino status did not have a direct relationship with 2015/2016 algebra scores, 
but it had an indirect relationship indicating that students of Hispanic or Latino origin were 
predicted to obtain scores that were on average 0.040 standard deviations lower than non- 
Hispanic students. White students had algebra scores in 2015/2016 that were 0.183 (0.145 
directly and 0.038 indirectly) standard deviation units higher than non-white students. Gender 
did not have a statistically significant relationship with 2015/2016 EOC scores. The number of 
absences in 2015/2016 was negatively related to algebra scores, with each standard deviation 
increase in absences corresponding to an expected 0.051 standard deviation decrease in algebra 
scores. 

The path analysis model in Equations 1 and 2 also revealed a significant positive 
relationship between 2014/2015 algebra scores and 2015/2016 algebra scores, indicating that a 
one standard deviation increase in scores in 2014/2015 is expected to correspond to a 0.274 


standard deviation increase in 2015/2016 scores, keeping all other variables constant. 


INSERT TABLES 2 AND 3 ABOUT HERE 


Logistic Regression Model of the Relationship between Algebra Nation Logins and Passing 
Rates of Algebra EOC Assessment Re-takers 
Table 4 shows the coefficients estimated with the logistic regression model with login 


indicators. Here we will focus on interpreting the odds for those coefficients that were 
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statistically significant. Positive relationships correspond to odds higher than one and indicate an 
increase in probability of passing the re-take of the EOC assessment. Negative relationships 
correspond to odds lower than one and indicate a decrease in probability of passing. In the 
presentation of results, we calculated 1/odds for those odds lower than 1, to present them in 
terms of an increase in the odds of failing the EOC assessment. The results show that students 
who logged in between 5 and 30 times during 2015/2016 were 1.220 times more likely to pass 
the EOC assessment in 2015/2016, after controlling for free/reduced lunch status, days absent, 
race, Hispanic origin, and gender. 

The logistic regression model also revealed that for each point increase in the EOC 
assessment in 2014/2015, the students were expected to become 1.029 times more likely to pass 
the re-take of the EOC in 2015/2016, after controlling for covariates. For each absence in 
2015/2016, the odds of failing the EOC in 2015/2016 was predicted to increase by 1.018 times. 
Students who were eligible for free/reduced lunch were 1.375 times more likely to fail the EOC 
in 2015/2016 than students who were not eligible for free/reduced lunch. White students were 


1.362 times more likely to pass the EOC in 2015/2016 than not-white students. 


INSERT TABLE 4 ABOUT HERE 


Path Analysis of the Relationship of Algebra Nation Video Views and Test Yourself 


Module Use with Algebra scores 
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The path analysis model in Equations 3 and 4 had exact fit to the data, as indicated by a 
non-significant chi-square test [i.e., X°(10, N = 3987) = 10.954, p = 0.361]. Also, close fit was 
supported by CFI = .998, TLI = .994, RMSEA = .005, and SRMR = .005. Given strong evidence 
that the path model fit the data well, we proceeded to interpret parameter estimates. 

Table 5 shows the coefficients estimated with Equation 3 of the path analysis model, 
which has video views and Test Yourself module use as predictors of 2014/2015 algebra scores. 
The results show that students who watched between 5 and 30 videos during 2014/2015 had 
algebra scores that were 0.127 standard deviation units higher on average than students who 
watched fewer than five videos, after controlling for free/reduced lunch eligibility, days absent, 
race, Hispanic origin, and gender. In addition, students who watched more than 30 videos during 
2014/2015 had algebra scores that were 0.126 standard deviation units higher on average than 
students who watched fewer than five videos. Table 5 also indicates that students who used Test 
Yourself module more than 30 times during 2014/2015 had algebra scores that were 0.229 
standard deviation units higher on average than students who used Test Yourself module fewer 
than five times. 

In 2014/2015, students of Hispanic or Latino origin had algebra scores that were 0.151 
standard deviation units lower than students not of Hispanic or Latino origin (see Table 5). 
Students who were eligible for free/reduced lunch had algebra scores in 2014/2015 that were 
0.176 standard deviations lower than students who were not eligible for free/reduced lunch. 
White students had algebra scores in 2014/2015 that were 0.136 standard deviations higher than 
non-white students. 

Table 6 shows the estimates of the coefficients for Equation 4 of the path analysis model. 


The results showed no difference between students who watched fewer than 5 videos and those 
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who watched between 5 and 30 videos in 2015/2016 on algebra scores, but it showed an indirect 
relationship between watching 5 to 30 videos in 2014/2015 and the 2015/2016 algebra scores. 
Specifically, students who watched between 5 and 30 videos in 2014/2015 had algebra scores in 
2015/2016 that were 0.035 standard deviations higher than the group watching less than five 
videos. Also, students who watched more than 30 videos during 2015/2016 had algebra scores 
that were 0.176 standard deviation units higher on average than students who watched fewer than 
five videos. Students who watched more than 30 videos in 2014/2015 had mean score in the 
2015/2016 assessment that was 0.034 standard deviations higher than students who watched 
fewer than 5 videos, through the mediation of the 2014/2015 algebra score. 

The results presented in Table 6 indicate that students who used the Test Yourself tool 
between 5 and 30 times during 2015/2016 had algebra scores that were 0.140 standard deviation 
units higher on average than students who watched fewer than five videos. Students who used 
the Test Yourself module more than 30 times in 2015/2016 had EOC scores that were 0.185 
standard deviation units higher on average than students who used the Test Yourself module 
fewer than five times. Also, students who used the Test Yourself module more than 30 times in 
2014/2015 had a higher mean algebra score in 2015/2016 by 0.062 standard deviations. 

As shown in Table 6, the model also revealed that a one standard deviation increase in 
algebra scores in 2014/2015 corresponded to a 0.273 standard deviation increase in scores on the 
2015/2016 EOC assessment retake, keeping all other variables constant. Furthermore, for each 
one standard deviation increase in student absences in 2015/2016, algebra scores were predicted 
to decrease by 0.051 standard deviations. Students of Hispanic or Latino origin had algebra 
scores in 2015/2016 that were 0.041 (indirectly) standard deviation units lower than students not 


of Hispanic or Latino origin, through the mediation of 2014/2015 EOC scores. Students eligible 
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for free/reduced lunch had scores in 2015/2016 that were 0.200 (0.152 directly and 0.048 
indirectly) standard deviation units lower than students who were not eligible for free/reduced 
lunch. White students had scores in 2015/2016 that were on average 0.176 (0.139 directly and 
0.037 indirectly) standard deviation units higher than non-white students. There were no 


differences between males and females in algebra scores in 2015/2016. 


INSERT TABLE 5 AND 6 ABOUT HERE 


Logistic Regression Model of the Relationship of Algebra Nation Video Views and Test- 
Yourself Use with Passing Rates of EOC Assessment Re-takers 

Table 7 shows the coefficients estimated with the logistic regression model with video 
views and Test Yourself module responses indicators. We will interpret the odds of the variables 
found to be statistically significant, inverting the negative odds. We found that students who used 
the Test Yourself module between 5 and 30 times during 2015/2016 were 1.590 times more likely 
to pass the EOC assessment in 2015/2016 than students who used Test Yourself less than five 
times, after controlling for algebra scores in 2014/2015, free/reduced lunch eligibility, days absent, 
race, Hispanic origin, and gender. Video views were not related to increases in the odds of passing 
the EOC assessment. The model also revealed that for each point increase in scores in 2014/2015, 
students were 1.029 times more likely to pass the 2015/2016 EOC assessment. For each absence a 
student had in 2015/2016, the student was 1.018 times more likely to fail the assessment. Students 


who were eligible for free/reduced lunch were 1.342 times more likely to fail the EOC in 
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2015/2016 than students who were not eligible for free/reduced lunch. White students were 1.343 


times more likely to pass the EOC assessment in 2015/2016 than not-white students. 


INSERT TABLE 7 ABOUT HERE 


Discussion and Conclusion 

Both the logins model and the video views and Test Yourself tool models indicated that 
in the 2014/2015 school year, students who used AN more were predicted to score higher on the 
EOC assessment than less-frequent users, even though they did not achieve a passing score on 
their first attempt (Research Question 1). However, the effects of logins were not consistent 
across the login groups, because students having between 5 and 30 logins had higher scores than 
students with fewer logins, but students with more than 30 logins did not (see Table 2). This 
inconsistency may be due to differences in power levels, because the group with more than 30 
logins is substantially smaller than the group with between 5 and 30 logins (see Table 1). For 
video views, the coefficients of the group with between 5 and 30 video views and the group with 
more than 30 video views are similar. For Test Yourself module responses, the effect was not 
significant for the group with between 5 and 30 uses but was significant for the group of more 
than 30 uses (see Table 5), indicating a pattern of increase in Algebra EOC scores as the number 
of Test Yourself module responses increases. 

In the next year (i.e., 2015/2016), the students who had failed the EOC assessment and 


were now preparing to re-take it and had more frequent AN use achieved higher scores than 
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comparable students with lower use (Research Question 2). There was a consistent increase in 
total effects (i.e., sum of direct and indirect effects shown in Table 3) from the students group 
with between 5 and 30 logins to the student group with more than 30 logins. With respect to 
video views and test yourself module, there was also a consistent increase in total effects from 
the group with between 5 and 30 uses from the group with more than 30 uses (see Table 6). 

AN usage predicted an increase in students’ scores from the previous year. Using AN 
more than 30 times in both 2014/2015 and 2015/2016 school years was associated with a more 
successful EOC retake outcome. The positive relationship was observed for two types of Algebra 
Nation usage: overall frequency of logging in, and using specific resources such as viewing 
instructional videos and completing practice problems. With regard to the activities within the 
AN platform that appeared to be most helpful (Research Question 3), there was some indication 
that higher use of the Test Yourself module in which students practiced solving problems similar 
to those that they would encounter on the actual exam had more impact than viewing 
instructional videos, because the standardized coefficients were larger by some extent (see Table 
6). Furthermore, in the logistic regression shown in Table 7, Test Yourself responses were 
related to increased odds of passing the exams, while video views were not. This is consistent 
with prior work showing that interactive online activities are more strongly linked with improved 
learning outcomes than viewing videos or reading text (Koedinger et al., 2015, 2016). Also, as 
OERs afford environments that require students’ use of self-regulatory strategies, the finding that 
use of the Test Yourself module is most strongly related to student achievement is consistent 
with self-regulated learning theory (Panke & Seufert, 2013, Kim, Lee, Park, in press). 

The analyses also indicated that performance on the EOC assessment was related to 


student characteristics. More specifically, not-white students, those of Hispanic background, and 
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those eligible for free/reduced lunch tended to have lower scores. Performance was also related 
to student behavior with respect to school attendance. Not surprisingly, more frequent school 
absence was associated with lower algebra scores. These patterns are generally consistent with 
other work on risk factors associated with algebra performance (Snipes & Finkelstein, 2015). 
The results indicate that struggling students with higher usage of AN scored better on the 
EOC assessment than comparable students with less (and very little) AN use. This pattern was 
apparent in both school years. Students with higher level of usage of AN were also more likely to 
achieve a passing score the following year. The results of this study show evidence of promise of 
AN as an OER to help struggling students learn algebra. Because OER characteristics and how 
they are used by students vary widely (Olcott Jr., 2012), it is necessary to connect the 
mechanisms associated with OER use to a specific theory of learning and cognition (Panke & 
Seufert, 2013) to predict how the results obtained with AN in this study may generalize to other 
OERs. For example, from a self-regulated learning perspective, it is expected that OERs such as 
AN will benefit the most students who are effective self-regulators, and the relationship between 
OER use and student achievement would replicate for OER that facilitate self-regulated learning. 
Kim, Lee and Park (in press) used data from several OER in the OER hub project to classify 
students according to self-regulated learning strategies. They found four classes that differed 
with respect to their extent of engagement with social and individual self-regulated learning 
strategies, but did not examine their relationships between these classes and learning outcomes. 
The current study provides a contribution by linking a specific self-regulated learning strategy 
(i.e. answer questions in the Test Yourself module) to student achievement. Future research is 


needed to connect a variety of types of self-regulated use of OER to student achievement. 
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This study contributes to the body of evidence about effects of OER student use on high- 
stakes educational outcome. However, the conclusions of this study are subject to the following 
limitations: Because the current study was not an experimental design that manipulated 
availability of videos and the Test Yourself module randomly and independently, conclusions 
about relative effectiveness of these AN activities are preliminary. Therefore, additional studies 
are needed to obtain stronger evidence for the “do-er effect, to understand which self-regulated 
learning strategies in AN are related to student achievement, and to design strategies that 
optimize self-regulated learning in AN. Experimental and quasi-experimental studies are 
necessary because it is possible that students who used AN more often did so for reasons that 
might have contributed to their higher algebra scores, such as they may have been more 
motivated to pass the EOC assessment, they may have had more encouragement from teachers or 
parents, and/or they may have had better guidance about how to use the site effectively. 
However, correlational studies form a necessary foundation to justify and encourage future 
research on effectiveness claims (Institute of Education Sciences, U.S. Department of Education, 
and National Science Foundation, 2013), and our study serves this important purpose. 
Additionally, the analysis focused on the population of students who had already failed and who 
were targeted because they were very likely to fail again. The sample included only students who 
had used Algebra Nation at least once because data was not available on non-users. Given the 
motivational and learning challenges associated with these students, it is striking that there was 
any detectable relationships between achievement and AN use. Also, for most students the 
difference was not substantial; indeed, for the majority, the improvement was not enough to 
ensure a passing score. The overall passing rate of this sample when they re-took the EOC 


assessment was 9.5%. The effect sizes found in this study are consistent with prior work 
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indicating that relatively small effect sizes are expected with the use of standardized test scores 
as the outcome measure (Cheung & Slavin, 2013). 

Another limitation is that the study did examine other ways that students may have 
prepared for re-taking the EOC assessment besides AN, such as enrollment in a credit recovery 
course. However, not all students who fail the EOC assessment can retake the entire algebra 
course, and even if they could, doing so would mean a lost opportunity to take a different course. 
Also, enrolling all algebra re-takers into algebra courses would likely be an unsustainable burden 
on the education system in most states. The rapid adoption of the AN platform within the state 
suggests that struggling students and their teachers were actively seeking a resource that might 
be helpful and that these stakeholders are willing to consider an alternative route to teaching 
struggling students. 

The present results converge with previous work in suggesting the potential value of open 
education resources. The next question is to identify how such resources can be most effectively 
used (Lowes & Lin, 2017). As a result of implementation research that established what 
elements teachers most valued and prompted changes to the platform, Khan Academy ultimately 
transitioned from an OER to being a formal part of the curriculum in a number of states and 
districts (Heppen et al., 2006; Phillips & Cohen, 2015). Similarly, in the case of AN, additional 
experimental research is needed to pinpoint factors such as the optimum frequency of use and the 
appropriate balance of activities such as watching instructional videos relative to doing practice 
problems as well as to determine its potential value for all students taking algebra, not only those 
who have a history of failure. The present study provided the necessary correlational foundations 
for this future intervention development work (Institute of Education Sciences, U.S. Department 


of Education, and National Science Foundation, 2013). 
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Table 1 


Sample sizes of groups included in the analysis 
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Logins 


Video views 


Test-yourself tool responses 


Gender 


Race 


Ethnicity 


Reduced/free lunch 


less than 5 times 
between 5 and 30 times 
more than 30 times 

less than 5 times 
between 5 and 30 times 
more than 30 times 

less than 5 times 
between 5 and 30 times 
more than 30 times 
Female students 

Male students 

White students 
Non-white students 


Hispanic/Latino origin 


Non-Hispanic/Latino origin 


Eligible 


Not Eligible 


Sample size Sample size 


2014/2015 


1442 


2104 


441 


1823 


1502 


662 


3393 


507 


87 


2209 


1778 


2065 


1922 


1267 


2720 


3141 


846 


2015/2016 


2069 


1720 


198 


2300 


1263 


424 


352) 


374 


86 


2209 


1778 


2065 


1922 


1267 


2720 


3141 


846 


Table 2 


Standardized Parameter estimates for path analysis model for the relationship between Algebra 


Nation Logins and Algebra I End-of-Course Assessment Scores in 2014/2015 
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Outcome: Algebra EOC Scores 2014/2015 Estimate S.E. P-Value 
Intercept 22.014 0.283 0.000*** 
Between 5 and 30 logins 0.086 0.037 0.018* 
More than 30 logins 0.071 0.055 0.195 
Days Absent -0.023 0.014 0.103 
Eligible for free/reduced lunch -0.179 0.039 0.000*** 
Hispanic/Latino origin -0.147 0.041 0.000*** 
White student 0.137 0.038 0.000*** 
Female student 0.045 0.032 0.156 
Residual Variance 0.985 0.004 0.000*** 


36 


Note. Statistically significant coefficients are in bold. Significant codes: *** 0.001; ** 0.01; * 


0.05. 


Table 3 
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Standardized Parameter estimates for path analysis model for the relationship between Algebra 


Nation Logins and Algebra I End-of-Course Assessment Scores in 2015/2016 


Outcome: Algebra EOC Scores 2015/2016 Relationship type Estimate S.E. P-Value 
Intercept 16.638 0.500 0.000*** 
Algebra EOC Scores 2014/2015 Direct 0.274 0.014 0.000*** 
Between 5 and 30 logins 2015/2016 Direct 0.093 0.032 0.004*** 
Between 5 and 30 logins 2014/2015 Indirect 0.024 0.010 0.020* 
More than 30 logins 2015/2016 Direct 0.195 0.076 0.010** 
More than 30 logins 2014/2015 Indirect 0.019 0.015 0.195 
Days Absent 2015/2016 Direct -0.051 0.016 0.002*** 
Days Absent 2014/2015 Indirect -0.001 0.000 0.106 
Eligible for free/reduced lunch Direct -0.157 0.042 0.000*** 
Indirect -0.049 0.011 0.000*** 
Hispanic/Latino origin Direct -0.002 0.037 0.948 
Indirect -0.040 0.012 0.001*** 
White student Direct 0.145 0.037 0.000*** 
Indirect 0.038 0.010 0.000*** 
Female student Direct -0.002 0.036 0.952 
Indirect 0.012 0.009 0.157 
Residual Variance 0.903 0.010 0.000*** 


Note. Indirect relationships are mediated by 2014/2015 algebra EOC Scores; Statistically 


significant coefficients are in bold. Significant codes: *** 0.001; ** 0.01; * 0.05. 
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Table 4 
Standardized Parameter estimates for Logistic Regression model for the relationship between 


Algebra Nation Logins and Algebra I End-of-Course Assessment Passing Rates in 2015/2016 


Predictor Estimate S.E.  P-Value Odds of passing 


Algebra EOC Scores 2014/2015 0.015 0.001 0.000*** 1.029 
Between 5 and 30 AN logins 0.102 0.045 0.021** 1.220 
More than 30 logins 0.172 0.108 0.112 1.398 
Days Absent -0.009 = 0.003 0.004** 0.982 


Eligible for free/reduced lunch ___-0.164 0.056 0.003** 0.727 


Hispanic/Latino origin 0.059 0.054 0.280 1.121 
White student 0.159 0.057 0.005** 1.362 
Female student -0.026 0.045 0.575 0.952 


Note. Statistically significant coefficients are in bold. Significant codes: *** 0.001; ** 0.01; * 


0.05. 


Table 5 
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Standardized Parameter estimates for path analysis model for the relationships between Algebra 


Nation Video Views, Test Yourself Module Responses, and Algebra EOC Scores in 2014/2015 


Outcome: Algebra EOC Scores 2014/2015 Estimate S.E.  P-Value 
Intercept 21.981 0.281 0.000*** 
Between 5 and 30 video views 0.127 0.039 0.001*** 
Between 5 and 30 Test Yourself module responses 0.059 0.053 0.263 
More than 30 video views 0.126 0.050 0.012* 
More than 30 Test Yourself module responses 0.229 0.090 0.011* 
Days Absent -0.018 0.014 0.212 
Eligible for free/reduced lunch -0.176 0.039 0.000*** 
Hispanic/Latino origin -0.151 0.041 0.000*** 
White student 0.136 0.038 0.000*** 
Female Student 0.044 0.032 0.166 
Residual Variance 0.980 0.005 0.000*** 


Note. Statistically significant coefficients are in bold. Significant codes: 


0.05. 


*** (2).001; ** 0.01; * 


Table 6 
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Standardized Parameter estimates for path analysis model for the relationships between Algebra 


Nation Video Views, Test Yourself Module Responses, and Algebra EOC Scores in 2015/2016 


Outcome: Algebra EOC Scores 2015/2016 Relationship Estimate S.E.  P-Value 
type 

Intercept 16.659 0.503 0.000*** 

Algebra EOC Scores 2014/2015 Direct 0.273 0.014 0.000*** 

Between 5 and 30 video views 2015/2016 Direct 0.027 0.038 0.473 

Between 5 and 30 video views 2014/2015 Indirect 0.035 0.011 0.001*** 

Between 5 and 30 Test Yourself module Direct 0.140 0.061 0.022* 

responses 2015/2016 

Between 5 and 30 Test Yourself module Indirect 0.016 0.014 0.264 

responses 2014/2015 

More than 30 video views 2015/2016 Direct 0.176 0.054 0.001*** 

More than 30 video views 2014/2015 Indirect 0.034 0.014 0.013* 

More than 30 Test Yourself module responses Direct 0.185 0.076 0.014* 

2015/2016 

More than 30 Test Yourself module responses Indirect 0.062 0.024 0.011* 

2014/2015 

Days Absent 2015/2016 Direct -0.051 0.016 0.002*** 

Days Absent 2014/2015 Indirect -0.005 0.004 0.215 

Eligible for free/reduced lunch Direct -0.152 0.042 0.000*** 
Indirect -0.048 0.011 0.000 


Hispanic/Latino origin Direct 
Indirect 

White student Direct 
Indirect 

Female Student Direct 
Indirect 


Residual Variance 
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-0.002 


-0.041 


0.139 


0.037 


-0.007 


0.012 


0.901 


0.037 


0.012 


0.036 


0.010 


0.037 


0.009 


0.010 
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0.950 


0.000*** 


0.000*** 


0.000*** 


0.857 


0.168 


0.000*** 


Note. Indirect relationships are mediated by 2014/2015 algebra EOC Scores; Statistically 


significant coefficients are in bold; Significant codes: *** 0.001; ** 0.01; * 0.05. 
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Table 7 
Standardized parameter estimates for Logistic Regression model for the relationship between 
Algebra Nation Video views and use of Test-yourself module and Algebra I End-of-Course 


Assessment Passing Rates in 2015/2016 


Predictor Estimate S.E.  P-Value Odds of 
passing 

EOC Scores 2014/2015 0.029 0.003 0.000*** 1.029 

Between 5 and 30 video views 0.038 0.095 0.691 1.038 

Between 5 and 30 Test Yourself module 0.464 0.145 0.001*** 1.590 

responses 

More than 30 video views 0,221 0.132 0.094 1.247 


More than 30 Test Yourself module responses 0.157 0.305 0.607 1.170 


Days Absent -0.018 0.006 0.004** 0.982 
Eligible for free/reduced lunch -0.295 0.110 0.007** 0.745 
Hispanic/Latino origin 0.113 0.105 0.284 1.119 
White student 0.295 0.110 0.007** 1.343 
Female Student -0.060 0.089 0.500 0.942 


Note. Statistically significant coefficients are in bold. Significant codes: *** 0.001; ** 0.01; * 


0.05. 
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Figure 1. Conceptual representation of the hypothesized relationships between Algebra Nation 


usage and Algebra I End-of-Course assessment scores 


